Queueing analysis of GPU-based inference servers with dynamic batching: A closed-form characterization

GPU-accelerated computing is a key technology to realize high-speed inference servers using deep neural networks (DNNs). An important characteristic of GPU-based that the computational efficiency, in terms processing speed and energy consumption, drastically increases by multiple jobs together batch. In this paper, we formulate as batch service queueing model with batch-size dependent times. We first show efficiency server monotonically arrival rate jobs, which suggests it energy-efficient operate under utilization level high possible within latency requirement jobs. then derive closed-form upper bound for mean latency, provides simple characterization performance. Through simulation numerical experiments, exact value well approximated bound. further compare curve measured real implementation performance explained derived formula.